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Critical Issues in Evaluating the Effectiveness of Technology 

The Secretary's Conference on Educational Technology- 1999 



Critical Issues in Evaluating the Effectiveness of 

Technology 

by 

Mary McNabb, North Central Regional Educational Laboratory 
Mark Hawkes, Dakota State University 
Ollik Rouk, Policy Studies Associates 



We are far enough along in the technological revolution and its application to learning that 
it is time for systematic review and analysis of what works best. 

U.S. Secretary of Education Richard W. Riley 
National Conference on Educational Technology 
Washington, D.C., July 12, 1999 

The Secretary's Conference on Educational Technology: Evaluating the Effectiveness of Technology on 
July 12-13, 1999, in Washington, D.C., noted a shift in schools* focus on technology. Where once the 
emphasis was on building and implementing a technology infrastructure, today it is on evaluating the 
effectiveness of its use in schools and classrooms. Parents and teachers, school boards and 
administrators, governors and state legislatures, and Congress all want to know if the nation's investment 
in technology is providing a return in student achievement. Indeed, if resources are to be expended on 
technology, it is becoming a political, economic, and public policy necessity to demonstrate its vital 
effectiveness. 

In his opening address, U.S. Secretary of Education Richard Riley remarked, "The primary reason for 
this conference is to gather information from all of the outstanding schools, districts, and states 
represented here-so that we can study it, share it, and learn from it. Just as important as learning what 
works, we must leam what does not work. We must not assume everything that employs technology is 
going to be successful. That is why evaluation is so important. And then we must use that evaluation to 
create positive change." 

The conference drew on the insights and collective wisdom of its attendees, starting with an emphasis on 
state-level technology evaluations. Evaluators from West Virginia explained how they isolated the 
effects of their Basic Skills/Computer Education initiative. They found that the more access to 
technology students had and the more their teachers believed that technology could help and were trained 
to use the technology, the higher students scored on the Stanford 9 (1 1% of the total gain scores). Idaho 
attendees described a four-year study focused on eight specific goals by which to evaluate the impact of 
the state's technology investment. Significant results included statewide academic gains as measured by 
the Iowa Test of Basic Skills (ITBS) and the Test of Academic Proficiency (TAP) for 8 th and 1 1 th 
graders. 

From there, the conference turned its spotlight on school practitioners who expressed the need for more 
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Critical Issues In Evaluating the Effectiveness of Technology 

formative evaluations than the summative evaluations described by state policymakers. Lively debates 
arose among teachers, district curriculum and technology coordinators, administrators, state curriculum 
and technology coordinators, state and national policymakers, and researchers in the evaluation of 
educational technology about ways of identifying and collecting technology evaluation data that is 
relevant at the local level but also useful for other stakeholders. 

During what Dale Mann of Interactive, Inc., called, "this developmental moment," conference 
participants exchanged promising evaluation strategies and techniques and considered how to respond to 
the many voices demanding to know technology's effects on schooling. The following seven critical 
issues in evaluating the (effectiveness of technology in education arose as a consequence of the 
interaction among stakeholders: 

• The effectiveness of technology is embedded in the effectiveness of other school improvement 
efforts. 

• Current practices for evaluating the impact of technology in education need broadening. 

• Standardized test scores offer limited formative information with which to drive the development 
of a school's technology program. Most schools are looking for additional means for collecting 
useful data for this purpose. 

• Schools must document and report their evaluation findings in ways that satisfy diverse 
stakeholders' need to know. 

• In order for evaluation efforts to provide stakeholder’s with answers to their questions about the 
effectiveness of technology in education, everyone must agree on a common language and 
standards of practice for measuring how schools achieve that end. 

• The role of teachers is crucial in evaluating the effectiveness of technology in schools, but the 
burden of proof is not solely theirs. 

• Implementing an innovation in schools can result in practice running before policy. Some existing 
policies need to be "transformed" to match the new needs of schools using technology. 

Critical Issue 1: The effectiveness of technology is embedded in the effectiveness of other school 
improvement efforts. 

Linking technology with core instructional objectives is what makes good, ejfective use of 
technology. That's the message we need to communicate. It's a process-not a number. 

Margaret Honey 

Center for Children and Technology 

A school's vision of the education it strives to provide students contains many elements, of which 
technology is but one. Other elements in this vision include administrative procedures, curricula, 
classroom organization, teachers' pedagogical approaches, time and space designations, 
school-community partnerships, and logistical and social factors. Developing ways to isolate the effects 
of technology within a dynamic environment where so many elements work together is one of 
evaluation's most challenging issues. 

Evaluators at the conference argued that social phenomena such as learning contain so many interacting 
factors that traditional experimental designs don't yield effective information. They support using 
evaluation designs that penetrate the effects of implementing technology at both individual, 
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Critical Issues in Evaluating the Effectiveness of Technology 

organizational, and sometimes even community levels. Evaluation designs of this type may be based on a 
system of learning benchmarks and other new forms of assessments that take the "localness" of 
evaluation into account. 

The high-stake decisions linked to technology implementation pressure educators to demonstrate that 
technology makes a difference in student learning. Many educators fear that evaluation places their 
technology programs at risk if they cannot produce measurable results in a relatively short time. The 
message that needs to be conveyed about the effectiveness of technology is that implementation of any 
sort produces outcomes. These outcomes, however, will be different at different stages of 
implementation. 

• At Mantua Elementary School in Virginia, technology is viewed not as an end in itself, but rather 
as a tool that augments the following four pillars of the Basic School, which are (1) the School as 
Community (bringing into focus how people relate to one another and work cooperatively to solve 
problems); (2) a Curriculum with Coherence (bringing an interdisciplinary approach to the 
acquisition of knowledge); (3) a Climate for Learning (providing the physical and motivational 
factors necessary for effective teaching and learning); and (4) a Commitment to Character 
(emphasizing how the school experience shapes the ethical and moral lives of children). The 
school community members use technology to simplify, facilitate, and enhance individualized and 
social learning processes within its interdisciplinary curriculum. Teachers are seen as leaders, 
facilitators, and mentors, well grounded in technology implementation strategies and well trained 
in the use of the most current computing equipment and software applications. Children exposed to 
interdisciplinary units of study use technology as a tool to become literate, cooperative, 
problem-solving, self-motivated learners and that is what Mantua is all about. What most 
distinguishes education at Mantua Elementary is that its students are not passive recipients of 
knowledge, but rather, active participants in die full educational process. 

• A technology-rich environment can support initiatives focused on improving learning outcomes as 
shown in Union City, New Jersey. The district framed its technology evaluation in conjunction 
with evaluation of school reforms such as students' development of literacy, higher-order thinking 
and collaboration skills. With district funds and iunds from Bell Atlantic and the National Science 
Foundation, technology became a key catalyst for school improvements that led to measurable 
academic achievements. But, as one of the most impoverished urban communities in the United 
States, Union City faced an uphill battle against state takeover with more than a plan to implement 
technology. Technology was just one of eight key reform strategies integral to the district's 
reported success in making school improvements. 



Critical Issue 2: Current practices for evaluating the impact of technology in education need 
broadening. 

To a certain extent, we are living out the decisions reflected in previous evaluation methods, 
which constrain our thinking about the purpose and effectiveness of technology in 
education . 



Walter Heinecke 

Curry School of Education at the University of Virginia 
The issue that confronts schools is broader than technology. It is about learning and the need to find new 
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• Critical issues in Evaluating the Effectiveness of Technology 

ways to identify and measure the skills and knowledge that students gain from using technology. It is 
about stakeholders' needs for information beyond self-report analyses and traditional standardized 
testing. It is about building the capacity of teachers to evaluate technology resources and to align their 
uses with the learning goals and content standards of the curriculum. It is about evaluating technology 
implementation efforts, curriculum integration methods, and learning processes in order to make sound 
decisions for continual improvement. Ultimately, the issue is about involving the key stakeholders, 
identifying appropriate measurable indicators, and developing reliable instruments that will yield 
insightful and valid information about what makes educational technology effective. 

The multimedia and networked capacities of the technology infrastructure are radically altering the face 
of technology-related practices in schools. However, the same rich diversity of technological tools that 
has created new learning opportunities for students has complicated the standardization of technology 
assessment. The fact that technology tools have undergone rapid cycles of innovation, causing constant 
change in the types of evaluation questions that need to be asked, compounds the difficulty even more. 
Educators, evaluators, and developers of measurement instruments struggle to keep current with the rush 
of information needs having to do with technology's effectiveness. 

In order to understand changing evaluation practices, stakeholders from the policy level on down to the 
home, need information on how using technology changes teaching and learning, its organizational 
impact, and the outcomes that can be reasonably expected at different stages of technology's 
implementation. In short, the challenge facing educators and evaluators is to compile enough evaluation 
data to substantiate and articulate technology's place in student and teacher learning. 

■ Many schools at the conference reported that they do use standardized tests as a part of their 
technology assessment program, but they also look at other outcomes. Educators in the Cherry 
Creek School District in Colorado, for instance, are using methods they call "far from refined." 
They evaluate progress based on district goals such as developing students' higher- order thinking 
skills, promoting collaboration among students working on projects, and honing the research skills 
of students around real-world topics. Instead of conducting quantitative research, they rely on best 
practices uncovered by currently published research to guide their technology implementation. The 
district's philosophy regarding evaluation is that isolating technology as the cause of achievement, 
productivity, or change is impossible. Therefore, they evaluate systemically: looking at SAT and 
ITBS scores related to programs in which technology is used; analyzing results from their 
Technology Integration/Student Achievement Specialist Survey; using the National Educational 
Technology Standards to develop ways of measuring student progress in technology foundational 
skills-to name a few of their multiple measures. 

■ Technology "ubiquity" in supporting other programs has convinced skeptics of its value. Schools 
at the conference suggested that were it not for the access to people, resources, and ideas that 
technology provides, school programs, from student peer-mentoring and summer enrichment to 
teacher professional development, would be seriously crippled. 

Critical Issue 3: Standardized test scores offer limited formative information with which to drive 
the development of a school's technology program. Most schools are looking for additional means 
for collecting useful data for this purpose. 

Who gave legislators reading and math test score to begin with? We did. We need to give 
them other measures, tell them how technology works, and help them see the results. 
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David Dwyer 
On-Track Learning, Inc. 

Standardized tests scores have become the accepted measure with which policymakers and the public 
gauge the benefits of educational investments. But educators and evaluation researchers argue that these 
scores say little about how to improve technology's effectiveness in schools. For this, they need 
information from formative evaluation. 

Formative evaluation tells what technology applications work, under what conditions, and with which 
students. It supplies information on how technology affects student attitudes toward learning. It can show 
the impact of technology on promoting collaboration among diverse learners. It can track technology 
literacy skills development and indicate the impact of technology access. Formative evaluation can tell 
teachers about their students' progress toward developing the skills to access, explore, and integrate 
information; think at high levels; and design, experiment, and model complex phenomena. 

Formative evaluation also yields information on the effectiveness of professional development activities, 
the adequacy of school management systems, and other issues having to do with building the school 
technology infrastructure. 

The good news is that schools have access to more information on these questions than they might think. 
Evidence of technology effectiveness may lie in fewer disciplinary referrals, students' completing more 
complex homework assignments, a new robustness in student performances, students taking more 
difficult electives or requesting particular teachers and courses, increases in requests for equipment and 
technical assistance, declines in special education placements, lower drop-out rates, rises in college 
applications and acceptances, increases in student job offers, and more parent participation. 

Other information collected through simple observations and questionnaires is formative as well. What 
technologies do teachers and students use and why? What is their attitude toward them? How has 
technology changed how teachers teach? How has it affected students' engagement with learning 
materials? Even the use of physical space, such as the rearrangement of study carrels in spaces where 
students can engage in learning with their peers, for example, can symbolize changes brought on by the 
use of technology. 

The problem is not so much the lack of data. The controversy revolves around accountability measures 
that ask the right evaluation questions; identify appropriate data sources; systemically capture the data; 
and analyze, interpret, and report the data in its appropriate context. 

• An educator from East Brunswick Public Schools in New Jersey maintained that of all the ways to 
evaluate technology integration, including hiring external consultants to conduct an evaluation, 
"the easiest to look at is standardized test scores." The most compelling evidence, however, is in 
what the district calls "secondary indicators." According to one of these indicators, when 
technology was integrated into a ninth-grade science curriculum, enrollment in chemistry classes 
swelled by nearly 500 percent, with overall enrollment in science courses growing by 17 percent. 

• Other educators at the conference reported discovering innovative indicators with which to 
evaluate technology's effectiveness: 

• High school humanities teachers in Oswego, New York, noted more varied citations in 
student papers after students began doing their research on the World Wide Web. 
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’ • The technology director in Montgomery, Alabama, observed that teachers put more detail 
and illustrative resources into their lesson plans than they used to. 

• Educators in Iowa used Bloom's taxonomy of cognitive learning as a guide to observing 
technology-integrated learning units. They found that technology-integrated learning 
reached higher in Bloom's hierarchy than nontechnology integrated learning. 

■ A count in several districts showed that interdisciplinary instruction was more prevalent in 
technology-supported instruction. 

■ A technology coordinator from Anderson County Schools, Tennessee, summed up 
technology's effectiveness this way: "I know there is great impact because if the file server 
drops, teachers want to call the buses and go home." 

• When test scores in Blackfoot School District, Idaho, revealed that students who used technology 
in their coursework scored 15 percent higher than those who did not use technology, no one in the 
community questioned technology's role or the capital investments that the district had made. Yet, 
when officials couldn't pinpoint a more exact effect of technology on student learning, they knew 
that their evaluation of technology effectiveness had to go deeper. They analyzed each piece of 
their technology system, including the role of learning benchmarks in content areas and grade 
levels, the usefulness of professional development activities, the unique effects of particular 
software, and the nature and goal of instructional activities. 

• Reporting students' achievement of core competencies on network technology has provoked new 
interest in school improvement in several communities. In almost every case, posting these 
competencies sparked districtwide debate about the relevance of present standards. These 
dialogues drove districts' examination of student achievement deeper than ever before, resulting in 
teachers being better informed and more committed to addressing agreed-upon competencies. 



Critical Issue 4: Schools must document and report their evaluation findings in ways that satisfy 
diverse stakeholders' need to know. 

We cannot survive on the random story anymore. 

Linda Roberts 

Office of Educational Technology at the U.S. Department of Education 



Interest in the effectiveness of technology is at an all time high. Parents want to know if children are 
developing a sound content base and thinking skills, and if they are going to be capable of lifelong 
learning in a fast-paced technological society. Teachers want to know if technology tools will help 
facilitate what they want to happen in the classroom. Administrators want to know if professional 
development activities are improving the way teachers use technology to teach. Funders, policymakers, 
and taxpayers want to know if technology is sufficiently promising to continue investing in it. 
Documenting and reporting evaluation data to meet these diverse stakeholders' need-to-know presents 
educational evaluators with a daunting series of challenges. 

The difference in the data needs of policymakers and educators is particularly acute. While policymakers 
want to see data on the isolated effects of technology, educators need information that is tied to systemic 
practices. Policymakers tend to value summative reports documenting student achievement while 
teachers and administrators value formative reports documenting implementation outcomes in order to 
make sound decisions about their technology plans. Many kinds of data are important, but each fails to 
satisfy the other. The best hope of closing this gap lies in helping all stakeholders to see (1) the 
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importance of technology as an effective component of the educational system, (2) how technology is 
and isn't capable of making a difference in curriculum and instruction, and 3) how innovative practices of 
teaching and learning with technology require multiple measures in order to verify its impact. 

Comparative language speaks loudly in this regard. It is useful to show technology's effects in a tangible 
way by, for example, comparing the instructional practices and learning opportunities that students have 
with technology to instruction without technology. Open dialogue and an understanding of mutual 
expectations for performance throughout the technology implementation process can resolve much of 
what differentiates stakeholders' interests in technology outcomes. What information do these groups 
need? What type and how much documentation do they require? What standards of documentation are 
most useful to different stakeholders? These are useful questions to consider. 

Finally, communicating about evaluation requires "speaking to" the stakeholder audience. What is the 
audience’s level of technology sophistication? How knowledgeable is the audience about evaluation 
terms and procedures? Speaking the language of the audience-by converting effect sizes into months of 
academic gain, for example-influences the way people think about technology and their support for it 1 . 

The technology infrastructure, itself, can be a useful tool for capturing, interpreting, and reporting data 
from multiple measures into understandable terms for a variety of stakeholder audiences. 

• When schools in the Juneau, Alaska, instituted electronic report cards to inform parents how and in 
what ways their children were meeting core content standards, communications between teachers 
and parents surged. A middle school teacher described this new type of access to parents as 
"empowering" her partnership with parents to guide their children's learning. 

• Educators at the conference considered parents one of their most important audiences. Once 
parents understood the value of technology, they became advocates. Parents, in fact, were often 
instrumental in moving technology into the classroom. The question that lingers is how to spread 
the message from parents to legislators. Some part of the answer, conference goers maintained, lies 
in encouraging parents to bear the message to policymaking bodies. 

• Conferees voiced concerns about the media's portrayal of technology programs in many of the 
nation's schools and districts. While applauding the media's role in informing the public about 
technology, educators charged that its interest in profiling technology growth and use "in one chart 
on one page" shortchanges the diversity of outcomes that technology produces. Participants 
suggested that the best way to encourage more comprehensive portrayals of school technology 
programs in the media is to link technology outcomes to goals that are deeply embedded in the 
mission and culture of the school. 



Critical Issue 5: In order for evaluation efforts to provide stakeholders with answers to their 
questions about the effectiveness of technology in education, everyone must agree on a common 
language and standards of practice for measuring how schools achieve that end. 

You have to show people the qualitative difference in what kids can actually do. 



Eva Baker 

National Center for Research on Evaluation, Standards and Student Testing at the University 
of California Los Angeles 

Dialog among stakeholders plays a central role in evaluation efforts. Stakeholders must be attuned to 
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common goals for the uses of technology, information needs, cultural terms, and methods for measuring 
outcomes. They must have consensus around roles and a clear vision of where they are going and the 
steps they need to take to get there. 

State-level consortia, made up of representatives from many stakeholder groups, can help develop 
guidelines that address schools' questions such as: What are important technology-induced indicators in 
our state and what instruments are available to measure these indicators? Where are the gaps in 
evaluation needs and measurement tools within our school communities? How can district educators and 
university researchers collaborate to develop evaluation instruments that will measure technology's 
effectiveness in our schools? 

Educators have known for a long time that technology can help students leam basic skills. But the tools 
that measure basic skills don't evaluate how technology supports students in developing capacities to 
think creatively and critically and vice versa. There is a need to develop additional evaluation tools that 
can help measure whether students are learning the "new basics" such as computer literacy, collaborative 
teamwork skills, and lifelong learning abilities. 

Left to themselves, schools have little time to develop and test such evaluation tools. While the 
successful evaluation of a school's technology does not necessarily require that researchers and 
evaluators be on the scene, seeking such expertise can be helpful, especially in evaluations that 
encompass several buildings or districts. Mary universities offer technology evaluation expertise. In 
addition, regional educational laboratories and technology education consortia allocate many of their 
resources to helping schools address evaluation issues. Other for-profit and not-for-profit organizations 
can also be helpful. Still, it is difficult for schools to identify what assistance is available. The field is ripe 
for developing scalable approaches, tools, and strategies for evaluating the effectiveness of educational 
technologies. 

The most useful tools yield information that is specific to the given student population and that allows 
teachers to track students' progress over time. Tools also need to measure those aspects and outcomes of 
learning that would otherwise be unattainable without the use of technology. Evaluation that 
demonstrates what students can do with technology that they couldn't do before access to the technology 
shows impact. For example, performance measures-observations of what students do and where they go 
on the Internet and how students collaborate with each other-help teachers track the impact of technology 
on student learning. Other measures that tap into education's broader curriculum aims include projects, 
essays, and extended performances. 

• Several school district representatives reported replacing student technology competency 
requirements with technology/content area integration standards as a basis for benchmarking 
grade-level technology integration. Their rationale was that this shift emphasizes technology's 
supportive role in teaching and learning rather than making technology use an end in itself. These 
educators believe that indicators articulating the components of a model instructional unit in 
fourth-grade science, for example, are more useful than technology competencies students should 
demonstrate at the fourth-grade level. 

• Early in the conference, it became clear that technology has spurred new terms and/or word 
meanings in our vocabulary. The term "engaged learning," for instance, had a very different 
meaning for participants from Chicago Public Schools than it did for educators in Cherokee 
County School District in Alabama. Similarly, when one participant referred to "performance 
standards," educators from New Hampshire's Campton School District envisioned a very different 
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set of standards than did their colleagues from the Okaloosa County School District in Florida. 
These exchanges illustrated the need to come to consensus on the terms and language of the 
evaluation process. Terms such as "technology integration," "benchmark," "core competency," 
"alternative assessment," and even "evaluation" and "student achievement" elicit different 
meanings from a range of educators and, unless they are made clear, can undermine evaluation 
efforts. 

Critical Issue 6: The role of the teacher is crucial in evaluating the effectiveness of technology in 
schools, but the burden of proof is not solely theirs. Evaluation is part of a reflective process. The 
more reflective we are, the more likely we are to improve our practice. 

Charol Shakeshaft 
Hofstra University 

Technology has revolutionalized what teachers do. It has added new breadth and depth to instruction. 
This, in turn, has transformed the role of the classroom teacher. In reformed educational settings, 
teachers guide students in using telecommunications to interact with astronauts in space, searching the 
Internet for up-to-the-minute information, and programming technology systems to help solve local or 
global problems. 

The countless hours teachers spend observing and interacting’with students makes teachers a rich source 
of data about the impact technology has on student learning. Teachers are the first to recognize increases 
in students' self-esteem and confidence, enhanced content area understanding, and more informed and 
empathic responses to world events as a result of using technology. This new role for teachers 
underscores the need for high-quality professional development in the use of technology and in 
determining what and how students leam best with technology tools. 

What teachers know about their students and about technology determines their competence in day-to 
day classroom decision-making. Good teachers evaluate their students and make decisions about how 
technology can boost their learning on a daily basis. Do students have access to the appropriate 
technology resources and tools? Are students using the technology efficiently? What kinds of learning 
tasks will challenge students' creative and critical thinking? In this new technology environment where 
there is not one instructional strategy but many, teachers need to know how to manage interactive group 
dynamics as well as technological systems. 

Professional development in schools that have implemented and evaluated educational technologies 
successfully helps teachers link effective uses of technology to impacts on student learning. Evidence of 
technology literacy, faculty meeting agendas, lesson plans, and classroom observations are all ways to 
determine a teacher's grasp of technology as a learning tool. The most useful program evaluation is one 
in which a strong formative element examines the connection between instructional practice, technology 
uses, and learning outcomes. 

Teachers are integral to the process of evaluating technology initiatives. They can act as partners with 
researchers to identify the sometimes very subtle impacts associated with technology uses. Teachers can 
also play key roles in measuring and documenting changes in student learning as they occur. Some of the 
best results m evaluating technology come from schools recognizing and harnessing the expertise 
teachers have in identifying technology-induced learning outcomes. 
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Teachers who have learned to use technology effectively in the classroom are convincing their colleagues 
of technology's potential Teachers training teachers to evaluate the usefulness of technology in the 
classroom remains a potent professional development strategy. 

• Evaluators must learn to trust teachers' ability to determine and describe technology's "ripple" 
effects, Margaret Honey, director of the Center for Children and Technology, explained during the 
conference. Success in studying school technology programs, according to Dr, Honey, often 
hinges on teachers contributing to the development of research questions and sharing ideas on how 
to record key indicators of effectiveness. 

• Lennox School District in southern California builds teachers' capacity to evaluate student learning 
with technology by having teachers collaboratively score students' work. Examining student 
products together builds consensus among teachers on the curriculum's core goals and the types of 
assessments that measure the achievement of those goals. 

• The nature of the questions that teachers ask their technology coordinators are data for evaluation. 
Their questions can indicate a school's position along the continuum of technology 
implementation. A technology coordinator from Helena, Montana, observed that when schools 
first deployed technology, teachers' questions centered on getting the hardware to work. Only a 
couple of years later, these same teachers' questions revolved around content and accessing 
resources through the network. 

Critical Issue 7: Implementing an innovation in schools can result in practice running before 
policy. Some existing policies need to be "transformed" to match the new needs of schools using 
technology. 

Our goal should be first, to understand the conditions of pro-social technology use and 
second to employ that understanding for learning improvement Both require more 
penetrating analysis than has heretofore been the standard. 

Dale Mann 
Interactive, Inc. 

Today's classrooms are expected to be technologically up to date. The same should be true for the 
policies that govern technology uses. When federal, state, or local district or building level policies do 
not keep up with classroom practices, innovative and effective practices can grind to a halt. To this end, 
educators have a leadership role in using evaluation information to shape the conversation around the 
kinds of policies that are most supportive in validating best practices that enhance the work of the school 
community. 

Policy issues rise to the surface around data. Who should have access to what data in the student 
information system? In theory, information about a student's family situation, for example, can help 
teachers understand and respond to student learning and behavioral problems. With today's information 
technology networks, accessing all kinds of personal family information in student files is possible-but 
what are the ethical policy implications for doing so? 

Still another example of how the lack of policies can slow down reform has to do with the equitable 
allocation of computers and other technology resources. Does a school distribute computers to students 
who need them the most, or to those students whose teachers show the most computer proficiency? What 
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i$ a school's responsibility for out-of-school computer access? How are scarce technology lab resources 
scheduled for use by the school community during and after school hours? Does a school have a policy 
governing the use of its technology to address adult technology literacy needs? 



Many school communities have recognized the need to create and enforce Internet usage policies, for 
instance, but what other less obvious technology-related policies are required to support and govern the 
best practices associated with implementing technology innovations into the school system? Local 
educators have the experience to help shape such questions and define successful practice for state and 
federal policymakers. These policymakers can then respond by developing policies that support the 
effective use of technology at the local level on a systemic basis. An important part of policy reform is to 
give policymakers a common language and data with which to speak to their constituents so that support 
for effective uses of technology will be widespread throughout the community. 

• Kayenta Unified School District (KUSD) is a small rural Navajo community located in an isolated 
region in the northeastern comer of Arizona, near the magnificent Monument Valley. This school 
district serves 2,600 students from Kayenta and several other smaller, more rural communities. 

The nearest public library is 100 miles away, while the nearest museums, bookstores, and 
universities are 150 miles. This isolation has provided the motivation to use technology to assist in 
increasing literacy, while permitting students to sustain critical elements of the rich traditional life 
of generations of Dine. After ten years of hard work building their technology infrastructure, 
KUSD presently has all six schools and administrative buildings, and all classrooms, offices, and 
administrators connected to an Intemet/intranet e-mail system. Determining ways to complement 
traditional instruction and community values with the global access provided by the Internet has 
been a compelling policy, as well as curricular, challenge faced by Kayenta and other rural isolated 
school districts. Kayenta distance learning policies opened the school community to the outside 
world in many ways. 

• More and more as teachers and parents gain access to e-mail communications and evaluation data 
via the Internet, schools are finding it difficult to maintain current information policies governing 
information access. A case in point is the use of e-mail for parent-teacher communication. While 
in theory, frequent communications between parents and teachers is a positive move forward, 
practitioners pointed out that having to respond to frequent requests from parents about their 
children's schoolwork tears them away from instructional planning time. This raises policy 
questions about teachers' obligations to respond to individualized e-e-mail requests from parents. 

• Educators at the conference demonstrated their broadened view of outcomes by recounting stories 
of timely access, attitudinal change, and increased motivation. Most initiatives in their early stages 
feature such stories. Their telling is an important step in shaping realistic public and legislative 
expectations for technology evaluation and supportive policies. 



What's Next? 



Schools that have partnered with other schools, universities, and educational service agencies to 
collaborate on technology planning, implementation, and research show compelling and productive 
applications of technology. Now comes the call for rigorous technology evaluation designs that are 
innovative and relevant to showing its impact. 



Researchers and educators are finding ways to partner in evaluating the technology initiatives that they've 
instituted. Such partnerships are revolving around many different purposes. Universities are partnering 
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with schools to construct the next generation of evaluation tools and processes. State and federal 
governments are beginning to reserve grant monies for evaluation activities in order to identify and 
disseminate information about technology practices that work and that may benefit schools in other 
contexts. Policies are beginning to be discussed that will support these and other innovative practices. 

The Secretary's Conference on Educational Technology: Evaluating the Effectiveness of Technology 
took a step forward in bringing together federal, state, and local evaluators with school practitioners to 
understand the many puzzle pieces involved in evaluating the effectiveness of technology in education. 
In many respects, the discussions and presentations at the conference raised more questions than they 
answered. In other respects, the diversity of research and best practices shared by participants represents 
the "state-of-the-field" in evaluating the effectiveness of technology in schools. The need to evaluate the 
effectiveness of technology in schools fuels a vast potential for collaboration among schools, 
universities, research organizations, businesses, and community groups. 

A multimedia CD-ROM will be available for those wishing to further delve into these critical issues 
raised at the conference (see the order form). In addition to providing examples of and expert 
commentary on spotlight school evaluation practices, it also contains templates for guiding school 
leaders' thinking about designing an evaluation plan. The U.S. Department of Education has launched a 
conference Web site at http://www.ed.gov/Technology/TechConf/1999/. Check it for the conference 
proceedings, announcements, and online events. In addition, follow-up regional conferences are being 
planned for the millennium year to focus more in-depth on the impact technology has on schooling and 
the evaluation needs to show the nature of those impacts. 
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